Tired of trying to tune VE/LTFTs? Let me help!

[redhardsupra]

Well I hope I'm not the only bloke out there feeling a little inadequate and wondering how the hell you run a "script" (what ever that is) Huh?

Inadequate? How do you think I felt when I couldn't find silly Fuel Trim Cell boundaries?
But on a serious note, this is Perl. Runs on Windows, Unix, and just about any other modern OS out there. Perfect for parsing large amounts of data, not perfect for number crunching, but in this case, we do one series of calculations, with <50k points, and that's no stretch for modern cpu's.

Ok, I understand the point as such and you are looking to clean out any possible irrelevant logged information but the process involved is a tad confusing.

The whole point is to have the least interactive process possible. While creating and recreating this stuff in Excel is kinda fun the first few times, as you get intimate with the process, after week 2 it all became repetitive and boring to me. I just wanna use computers in a manner they were created for. Write a program for the manual procedure, and just keep feeding it new data to obtain near-instant results. Isn't that what we have computers for, not 'bitching about movies and swapping pornography' ('jay and silent bob strike back')?

Most of us can plug data into excel, that fine but perhaps you could explain how we implement this so called "script" on our logged data.

perl vehist.pl yourdatalog1.csv

You're most probably in windows, so you can get a very nice, legally free Perl from http://www.activestate.com/Products/ActivePerl/ .

I am more than willing to help out anyone to use this script, this is a good direction to go with I think. Granted, this is more of a proof of concept thing, which hopefully eventually will end up fully integrated into HPT (discount for creators maybe? ) but it should beats the hell out of the Excel way going about this whole process.

I am willing to contribute code, I am not a programmer however, so doing things like modularizing this code is a bit outside my skill range. I can, however, write the basic nuts'n'bolts code, and verify its correctness against other ways of dealing with it (different code, Excel, by hand calculation, etc).

I'm gonna start implementing little things, like the half-resolution output, hopefully that will get the momentum going.

[torchredfrc]

Comparison of results:
this is what I got by summing up Average values from the HPT scanner histogram:

   0.4   0.8   1.2   1.6   2   2.4   2.8   3.2   3.6   4   4.4
15   0   0   -11   0   -8   -6   -5   -2   0   0   0
20   0   0   -11   -9   -11   -6   -2   -3   -5   0   0
25   0   -15   -13   -11   -9   -3   0   -4   -1   0   0
30   0   -8   -11   -3   -5   -1   0   2   2   -5   0
35   0   -8   -6   -1   -3   2   3   2   5   0   0
40   0   -5   -1   1   -1   2   1   3   0   0   1
45   0   -4   1   2   1   1   -2   1   0   -3   0
50   0   -1   2   4   3   2   -3   3   0   -2   1
55   0   2   5   7   6   4   0   1   -2   1   0
60   0   5   7   8   6   5   3   0   2   0   0
65   0   6   7   9   4   4   5   1   0   0   0
70   0   0   8   8   6   4   4   4   0   0   2
75   0   0   8   8   6   5   5   5   0   4   0
80   0   0   7   5   4   5   3   4   3   3   0
85   0   0   3   4   4   6   3   4   2   5   5
90   0   0   7   5   4   6   4   4   5   5   0
95   0   0   6   4   3   4   2   1   0   0   0

This is the result of what I got by running your script (with my mods) on the same log:

   0.4   0.8   1.2   1.6   2   2.4   2.8   3.2   3.6   4   4.4
15   0   0   0   0   -15   -14   -4   0   -6   0   0
20   0   0   -14   -12   -11   -9   0   0   -6   0   0
25   0   0   -12   -13   -6   -5   0   0   -3   0   0
30   0   -9   -10   -7   -4   -4   -1   0   -4   0   -5
35   0   -7   -8   -2   -1   -1   2   3   3   0   0
40   0   -6   -5   0   1   -1   1   1   1   0   0
45   0   -3   1   2   3   0   3   -4   0   -3   0
50   0   -1   1   4   5   3   2   0   2   0   0
55   0   0   5   7   8   4   3   3   -2   0   0
60   0   0   6   7   7   5   4   4   -4   0   0
65   0   0   9   7   8   4   3   9   -2   0   0
70   0   0   0   8   7   5   3   4   0   0   0
75   0   0   0   7   6   4   3   5   0   0   0
80   0   0   0   6   5   4   4   9   2   0   0
85   0   0   0   2   4   4   5   3   7   0   0
90   0   0   0   1   3   5   5   4   0   0   0
95   0   0   0   3   3   3   3   5   0   0   0

Quite different, ain't it?
That means few things:
1. Different criteria for admitting/tossing out values (you just take anything > -100 and < 100) maybe we should be more selective, like +/- 15?
2. Ignoring data with other, certain conditions (coolant not up to full temp, idling in traffic, etc). This one is easy to verify, if we gather exclusively RPM,MAP,STFT's,LTFT's then their averages should be closer to ours
3. Resolution between values accounted and values displayed. We don't see any decimals in the Histogram, but we got them in the logs. (the conclusions of 'Can a butterfly cause a tornado' paper come to mind)
4. Averages might be counted differently. Maybe they're weighted toward the newer values, as most of us just reset the fuel trims and drive for an hour, don't do 5 full driving cycles in variety in conditions (aka proper scientific approach, not just a impatient hack)

Any other things that can go wrong that I haven't accounted for?

Interesting. I found my results corresponded at first glance. Maybe I didn't pay close enough attention though. My script doesn't seems to either be leaving out valid data or the HPTuners software is doing extra calculations. I could be curious to see the actual math behind the HPTuners software and their cirteria (if any) to throw out invalid data. If anything, there should be less zeros in the compiled results table (unless you are just using one log at a time to compare results). If you do just use one log at a time, you should see farily close to the same numbers... I believe the HPTuners numbers will be smaller though because I suspect they do not drop the all 0 fields in their averages. Obviously those will make your numbers less than what you will get using my script.

What type of data cable are you using? I believe I'm only seeing the +/- 100's because I'm using a USB to RS232 cable.

I'm glad we got together on this today, I'm in the process of getting with the Perl genius I know to get a module worked up to read in the csv files in a more abstract manner.

Is there any chance HPTuners could put together some software to put csv files back in to the HPTuners binary format? I would like to see the results the HPTuners histogram would produce with a longer data set.

[torchredfrc]

Well I hope I'm not the only bloke out there feeling a little inadequate and wondering how the hell you run a "script" (what ever that is) :huh2:

Ok, I understand the point as such and you are looking to clean out any possible irrelevant logged information but the process involved is a tad confusing.

Most of us can plug data into excel, that fine but perhaps you could explain how we implement this so called "script" on our logged data.

It is completely understandable that you don't know what a script is. Scripts are a sort of old technology and the are (mostly) used at a command prompt (unix / dos types systems). I am willing to help anyone who asks figure out how to run this script and interpret it with thier data set (right now, as redhardsupra pointed out, the script will only work if you do not log any extra data other than the HPTuners defaul).

My post shows some examples of how it works (picture of the command prompt on Windows XP) and with the use of the Excel file I made (you'll have to modify it as I only used half resolution - 99 VCM) you should be able to get a good idea of what you need to do to your VE table using a more scientific approach than just a run-by-run hack.

[redhardsupra]

any progress guys?

[torchredfrc]

I havn't had time to work on the module. I think I've found another bug in the initial script and I've been trying to pin point it.

Nice work on the MAF sheet though. Have you considered doing it out of Excel so you can process more data points? (combined logs like what I am trying to accomplish here)

[redhardsupra]

Already in the works, but i'd like to use your generic log reading module to do it, that's why I was asking.

[torchredfrc]

makes sense

[marc_w]

As a side note to this:

Anyone have any tips or tricks on how to weed out the spikes on a system running a regular ol' com-port setup?

I get a lot of them. Most of the time they are the same data fields that get them - like the FT's. Every once in a while it's different things like KR.

[torchredfrc]

As a side note to this:

Anyone have any tips or tricks on how to weed out the spikes on a system running a regular ol' com-port setup?

I get a lot of them. Most of the time they are the same data fields that get them - like the FT's. Every once in a while it's different things like KR.

Can you post a sample from your log file and point out the spikes you are talking about? I've only noticed the +/- 100 spikes with my setup.

[torchredfrc]

Already in the works, but i'd like to use your generic log reading module to do it, that's why I was asking.

Is this list complete?

Current Firing Cylinder
Engine Coolant Temp (SAE) (°F,°C)
Engine Oil Life (%)
Engine Oil Pressure (psi,kPa)
Engine RPM (SAE) (rpm)
Engine Run Time (sec)
ETC Desired Position (%)
ETC Pedal Average Position (%)
ETC Pedal Position (%)
ETC Position (%)
Generator F Terminal (%)
Ignition Voltage (V)
Intake Air Temp (SAE) (°F,°C)
Startup Engine Coolant (°F,°C)
Throttle Position (%)
Throttle Position (V)
Throttle Position (SAE) (%)
Time Since Engine Start (SAE) (sec)
IAC Desired Position
IAC Position
Idle AC Airlfow (lb/min,g/sec)
Idle Adapt (STIT) (lb/min,g/sec)
Idle Base Gear Airflow (lb/min,g/sec)
Idle Base PN Airflow (lb/min,g/sec)
Idle Base Sum Airflow (lb/min,g/sec)
Idle Cracker Airflow (lb/min,g/sec)
Idle Desired Airflow (lb/min,g/sec)
Idle Fans Airflow (lb/min,g/sec)
Idle Follower Airflow (lb/min,g/sec)
Idle Park Position Airflow (lb/min,g/sec)
Idle Startup Airflow (lb/min,g/sec)
Idle Startup Retard Airflow (lb/min,g/sec)
LTIT Gear/ACoff (lb/min,g/sec)
LTIT PN/ACoff (lb/min,g/sec)
BARO Sensor (psi,kPa)
Barometric Pressure (SAE) (psi,kPa)
Boost Control Level
Boost Solenoid Control (%)
Dynamic Airflow (lb/min,g/sec)
Dyanmic Cylinder Air (g/cyl)
Manifold Absolute Pressure (SAE) (psi,kPa)
Manifold Air Pressure (V)
Manifold Air Pressure - 2 bar (psi,kPa)
Manifold Air Pressure - 3 bar (psi,kPa)
Manifold Vacuum (psi,kPa)
Mass Air Flow (Hz)
Mass Air Flow (lb/min,g/sec)
Mass Air Flow (Freq Error) (lb/min,g/sec)
Mass Air Flow (SAE) (lb/min,g/sec)
Volumentric Efficiency (%)
AFR Advance (°)
Base Advance (°)
Coolant Advance (°)
Cranking Advance (°)
EGR Advance (°)
IAT Advance (°)
Ignition Timing Advance (SAE) (°)
Smoothing Advance (°)
Torque Mgt Advance (°)
Burst Knock Retard (°)
Knock Learn Factor
Knock Retard (°)
Knock Sensor Voltage In Range - Front
Knock Sensor Voltage In Range - Rear
Knock Sensor Voltage Out of Range - Front
Knock Sensor Voltage Out of Range - Rear
Startup Retard (°)
Absolute Load (SAE) (%)
Calculated Load (SAE) (%)
Delivered Engine Torque (ft/lb,N/m)
Desired Engine Torque (ft/lb,N/m)
Commanded EGR (SAE) (%)
Commanded EVAP Purge (SAE) (%)
Commanded Secondary Air (SAE)
EGR Desired (%)
EGR Error (SAE) (%)
EGR Position (%)
EGR Position (V)
EVAP Purge (%)
Cycles of Misfire Data
Misfire Current Cylinder #1
Misfire Current Cylinder #2
Misfire Current Cylinder #3
Misfire Current Cylinder #4
Misfire Current Cylinder #5
Misfire Current Cylinder #6
Misfire Current Cylinder #7
Misfire Current Cylinder #8
Misfire Cycle Delay Counter
Misfire Cylinder Mode Index
Misfire Cylinder MOde Index Level
Misfire Failures
Misfire History Cylinder #1
Misfire History Cylinder #2
Misfire History Cylinder #3
Misfire History Cylinder #4
Misfire History Cylinder #5
Misfire History Cylinder #6
Misfire History Cylinder #7
Misfire History Cylinder #8
Misifre Passes
Misfire Revolution Mode Index
Misfire Revolution Mode Index Level
Misfire RPM (rpm)
Misifre/Abnormal Ratio
Total Misfires
Fail Counter
Mileage Since First Failure (mi,km)
Mileage Since Last Code Clear (mi,km)
Mileage Since Last Failure (mi,km)
Not Run Counter
Pass Counter
Warm up cycles w/ an emission fault
Warm up cycles w/o a non-emission fault
Cam Signal Hi to Lo
Cam Signal Lo to Hi
Crank Sensor Current Cyl
ODB Requirements (SAE)
Trans Calculated Gear Ratio
Trans Current Gear
Trans Fluid Temp (°F,°C)
Trans Fluid Temp (V)
Trans Input Shaft RPM (rpm)
Trans Output Shaft RPM (rpm)
Trans Turbine RPM (rpm)
Downshift Solenoid 3-2 PWM Duty Cycle (%)
Force Motor (PCS) Actual Current (amp)
Force MOtor (PCS) Desired Current (amp)
Force Motor (PCS) Duty Cycle (%)
Force Motor (PCS) Line Pressure (%)
Shift Time Error for 1-2 Shift (sec)
Shift Time Error for 2-3 Shift (sec)
Shift Time Error for 3-4 Shift (sec)
Time of Latest Shift (sec)
Trans Performance MOde
Trans Shift Mode
TCC Apply Time (sec)
TCC Mode
TCC PWM Duty Cycle (%)
TCC Slip (%)
TCC Slip RPM (rpm)
Delivered Trans Torque (ft/lb,N/m)
Traction Control Mode
O2 Avg. Bias Voltage B1S1 (mV)
O2 Avg. Bias Voltage B1S2 (mV)
O2 Avg. Bias Voltage B2S1 (mV)
O2 Avg. Bias Voltage B2S2 (mV)
O2 LR Avg. Transition Time B1S1 (sec)
O2 LR Avg. Transition Time B2S1 (sec)
O2 LR Transitions B1S1
O2 LR Transitions B2S1
O2 RL Avg. Transition Time B1S1 (sec)
O2 RL Avg. Transition Time B2S1 (sec)
O2 RL Transitions B2S2
O2 RL-LR Transition Ratio B1S1
O2 RL-LR Transition Ratio B2S1
O2 Time to Activity B1S1 (sec)
O2 Time to Activity B1S2 (sec)
O2 Time to Activity B2S1 (sec)
O2 Time to Activity B2S2 (sec)
O2 Voltage B1S1 (mV)
O2 Voltage B1S2 (mV)
O2 Voltage B2S1 (mV)
O2 Voltage B2S2 (mV)
Report O2 Sensor Configuration (SAE)
INJ. Pulse Width Avg. Bank 1 (msec)
INJ. Pulse Width Avg. Bank 2 (msec)
Injector Flow Rate (g/sec)
Fuel Trim Cell
Long Term Fuel Trim Bank 1 (SAE) (%)
Long Term Fuel Trim Bank 1 Avg. (%)
Long Term Fuel Trim Bank 2 (SAE) (%)
Long Term Fuel Trim Bank 2 Avg. (%)
Short Term Fuel Trim Bank 1 (SAE) (%)
Short Term Fuel Trim Bank 1 Avg. (%)
Short Term Fuel Trim Bank 2 (SAE) (%)
Short Term Fuel Trim Bank 2 Avg. (%)
Fuel Level Input (SAE) (%)
Fuel Tank 2 Level Sensor (V)
Fuel Tank Capacity (gal,L)
Fuel Tank Level (%)
Fuel Tank Level (gal,L)
Fuel Tank Level Sensor (V)
Fuel Tank Pressure (V)
Air Fuel Ratio (afr)
Air Fuel Ratio Commanded (afr)
Fuel Air Multiplier
Fuel System Status (SAE)
AC Pressure (V)
Fan's
VATS
PRNDL Status
Vehicle Speed (SAE) (mph,km/h)
Wheel Accel/Decel

If this is a complete list of everything you can choose to log (and I hope it is), I'll get something going on the module code for reading in the log file. I've got some good help on this and hopefully he can come up with a module fairly quickly.

[redhardsupra]

i don't know if it's complete, but it's beautiful

[torchredfrc]

i don't know if it's complete, but it's beautiful

I just spoke with my friend who is going to be working on the module. He said he didn't think it would be to difficult so I sent him the list. I'll let you know what he says after he sees how many options he will have to account for.